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EXECITTIVE  SUMMARY 

All  Research  &  Development  projects/programs  differ  in 
purpose.  The  future  impact  of  R&D  efforts  is  difficult  to 
determine  and  often  there  are  no  usable  data  for  evaluation  since 
there  is  no  generally  accepted  or  universally  applicable  method 
for  predicting  a  program's  worth.  Being  able  to  evaluate  the 
future  impact  of  R&D  programs  could  provide  useful  information  to 
many  people.  Principal  Investigators  could  use  this  information 
to  "sell"  their  projects,  and  project  sponsors  could  use  this 
information  to  evaluate  programs  and  make  important  decisions 
about  funding,  acquisition,  implementation,  etc. 

The  NAVTRASYSCEN  Training  Analysis  and  Evaluation  Department 
was  requested  to  develop  means  to  assess  the  impact  of  R&D 
programs  on  the  Navy. 

Training  effectiveness  estimates  and  cost  estimates  were 
collected  from  four  NAVTRASYSCEN  R&D  programs  using  the  Device 
Effectiveness  Forecasting  Technique  (DEFT) ,  the  Research  and 
Development  Impact  Analysis  (RDIA) ,  and  a  standard  life  cycle 
cost  analysis. 

The  collected  data  indicated  that  the  four  R&D  programs 
would  be  training  effective.  Moreover,  the  cost  estimates  for 
three  of  the  four  programs  were  lower  than  an  alternative 
training  system.  A  general  methodology  is  recommended  for 
performing  an  impact  analysis,  and  guidelines  for  determining 
appropriate  instruments  for  collecting  data  are  proposed. 

The  nature  of  R&D  is  dynamic  and  changing.  Therefore,  the 
data  presented  here  (gathered  May  -  June  1988)  may  be  subject  to 
refinement. 
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INTRODUCTION 

The  annual  Science  and  Technology  Review  gives  the  Naval 
Training  Systems  Center  (NAVTRASYSCEN)  Research  and  Development 
(R&D)  Department  (Code  7)  the  opportunity  to  present  ongoing 
projects/programs  to  their  sponsors  or  potential  sponsors.  An 
action  item  originated  from  the  1987  review  concerning  the  need 
for  inclusion  in  the  presentations  of  R&D  projects  an  estimate  of 
the  likely  impact  of  the  work  in  terms  of  cost  and  effect. 
Subsequently,  Code  7  requested  the  Training  Analysis  and 
Evaluation  Depairtment  (Code  1)  of  the  NAVTRASYSCEN  to  develop 
an  approach  and  guidance  to  assess  the  impact  of  NAVTRASYSCEN  R&D 
projects.  The  approach  was  to  include  a  methodology  that  could 
be  applied  to  each  R&D  program  to  produce  estimates  of  cost  and 
effectiveness.  Not  only  would  these  estimates  predict  potential 
impact  on  the  Navy,  but  they  would  give  the  principal 
investigators  additional  information  for  "selling”  their 
projects/programs.  Moreover,  the  estimates  would  provide  data 
for  the  sponsors  and  potential  end-users  to  base  decisions;  e.g., 
whether  to  continue  funding  a  program  or  whether  to  increase 
funding  in  order  to  modify  a  program. 

A  straightforward  rationale  such  as  that  represented  in 
Table  1  could  then  be  applied.  For  example,  if  the  training 
effectiveness  of  the  alternatives  is  different  and  the  life  cycle 
costs  are  equal,  the  rule  states  to  select  the  alternative  with 
the  highest  training  effectiveness  estimate  (note:  an  existing 
training  system  could  be  an  alternative) .  Although  this 
rationale  appears  simple,  acquiring  the  means  to  make  rational 
selections  is  much  more  difficult.  The  difficulty  stems  from  the 
need  to  make  decisions  at  times  when  the  kind  of  information 
necessary  for  making  the  decisions  is  not  available  or  of  dubious 
quality. 

Figure  1  graphically  depicts  a  time  cycle  for  R&D  programs. 
Early  on  in  the  cycle  various  alternatives  are  proposed  which 
will  make  a  positive  change  in  the  status  quo.  Once  an  initial 
funding  decision  is  made  for  an  alternative,  the  R&D  program 
crosses  over  into  an  area  in  which  decisions  about  the  product 
will  not  necessarily  be  made  relative  to  the  status  quo. 
Decisions  within  this  area  are  usually  concerned  with  the 
program's  potential  impact  or  future  worth.  This  report  deals 
with  R&D  programs  which  fall  somewhere  in  this  area,  i.e., 
between  the  initial  funding  decision  and  the  implementation 
decision.  Consequent  to  this  time  frame  is  the  availability  of 
useful  data.  As  an  R&D  program  moves  from  the  initial  funding 
decision  to  the  stage  in  which  a  decision  can  be  made  to 
implement  the  product,  more  useful  data  should  become  available. 
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Table  1 

Decision  Rules  for  Choosing  Alternatives 


Training  Effectiveness  Life  Cycle  Cost 

of  Alternatives  of  Alternatives  Rule 


Different  Equal  Select  alternative 

with  highest 

training 

effectiveness 

Equal  Different  Select  alternative 

with  lowest  life 
cycle  cost 

Different  Different  Select  alternative 

with  higher  training 
effectiveness  and 
lower  life  cycle  cost 

None  -  subjective 
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Additionally,  after  an  implementation  decision  has  been  made, 
then  subsequent  decisions  may  be  compared  with  the  status  quo; 
e.q.,  how  does  the  finalized  product  compare  with  that  which  is 
already  in  existence? 

While  presenting  the  background,  the  timing  of  decisions 
and  information  availability  are  further  discussed.  Also, 
presented  is  a  justification  for  using  analytic  techniques  versus 
empirical  approaches  to  gain  necessary  information. 

BACKGROUND 

Rose,  Wheaton,  and  Yates  (1985)  pointed  out  that  ultimately 
the  purpose  of  decisions  during  device  design  and  development  is 
to  insure  that  an  initial  and  sometimes  vague  training  concept  is 
translated  into  cost-effective  training  equipment  which  will 
eventually  be  placed  in  the  field  or  at  a  school.  Accordingly, 
they  noted,  decisions  can  be  based  on  progressively  better  data; 
e.g.,  more  detailed  and  precise  information  about  the  training 
requirement  becomes  available,  the  physical  and  functional 
characteristics  of  the  training  system  which  satisfies  the 
requirement  become  less  abstract,  and  information  about  how  the 
system  will  be  used  becomes  better  understood.  Consequently, 
estimates  of  system  effectiveness  and  cost  should  become  more 
concrete  well  after  the  initial  concept  formulation. 

During  training  system  design  and  development,  empirical 
data  are  not  usually  available  for  making  decisions.  Empirical 
evaluations,  when  conducted,  usually  take  place  after  the 
training  device  is  ready  for  training  (RFT) .  Many  R&D  efforts 
are  only  concepts  and  demonstrations  with  no  specific  application 
in  mind,  thus  making  it  nearly  impossible  to  conduct  empirical 
evaluations  that  are  germaine  to  a  particular  training  setting. 
Moreover,  when  empirical  data  can  be  gathered,  one  roust  be  sure 
the  methodology  includes  carefully  designed  and  well-controlled 
experiments  (Pfeiffer,  1987) .  These  experiments  can  be  costly 
and  difficult  to  administer.  Similarly,  decisions  made  during 
R&D  follow  the  same  limitations;  Cordell  and  Nutter  (1983) 
concluded  that  the  most  appropriate  methodology  during  R&D,  prior 
to  the  device  being  RFT,  involves  using  analytic  techniques. 

As  an  aside,  a  very  important  point  is  that  prior  to  the 
principal  investigator's  efforts  to  collect  analytic  data,  the 
principal  investigator  should  determine  if  there  are  any  similar 
products  already  in  use.  If  so,  data  from  a  similar  product  can 
be  collected  and  used  to  base  estimates.  Although  the  chances  of 
finding  a  similar  product  may  be  slim,  if  a  similar  product  can 
be  found,  this  avenue,  i.e.,  reasoning  by  analogy,  may  save  the 
principal  investigator  time  and  money.  Furthermore,  data  from 
actual  products  may  produce  better  estimates. 
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ORGANIZATION  OP  THE  REPORT 

The  remainder  of  the  report  addresses  analytic  and  cost 
estimation  techniques  for  assessing  the  impact  of  R&D  projects/ 
programs.  The  next  section  describes  two  training  effectiveness 
forecasting  models  which  were  chosen  for  their  applicability  and 
flexibility.  Also  included  in  this  section  is  an  explanation  of 
an  approach  for  cost  determination.  Following  that  is  a  brief 
description  of  the  four  selected  R&D  projects  along  with  results 
of  applying  the  proposed  approach.  Finally,  a  summary  of  the 
data  and  guidelines  for  administering  an  impact  analysis  is 
provided.  Appendices  A  and  B  contain  copies  of  the  instruments 
which  were  used  to  obtain  impact  estimates  for  the  four  selected 
R&D  projects. 
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RECOMMENDED  APPROACHES 

The  reader  should  recall  that  the  decision  rules  for 
choosing  among  alternatives  (Table  1)  require  an  assessment  of 
effectiveness  and  cost.  This  section  contains  some  recommended, 
general  approaches  to  the  estimation  of  effectiveness  and  cost  of 
training  R&D  efforts.  It  should  be  emphasized  that  the 
approaches  discussed  appear  reasonable  and  suitable  for  use; 
however,  the  individual  R&O  project  will  necessitate  some 
modification  or  tailoring  of  method.  It  is  the  intent  of  this 
section  and  the  examples  to  provide  the  reader  with  sufficient 
structure  and  insight  to  adapt  the  approach  to  the  specifics  of  a 
particular  R&D  impact  assessment  situation. 

Rose,  Wheaton,  and  Yates  (1985)  suggested  that  an  ideal 
methodology  for  analytically  evaluating  (or  forecasting)  the 
effectiveness  of  a  training  device  (readers  may  substitute 
training  R&D  product  for  device)  would  have  several  properties. 
The  methodology  would  be: 

o  applicable  at  different  stages  of  device  design  and 
development ; 
o  diagnostic; 
o  easy  to  use; 

o  supportive  for  different  levels  and  types  of  decisions. 

The  Device  Effectiveness  Forecasting  Technique  (DEFT)  and  the 
Multi-attribute  Utility  (MAUT)  method  are  two  analytic  methods 
which  meet  these  properties  and,  therefore,  were  examined  as 
potential  candidates  to  apply  to  four  R&D  programs/projects 
selected  by  the  NAVTRASYSCEN  R&D  Department.  Both  models  had  to 
be  modified  in  order  to  apply  to  the  four  projects.  The  MAUT 
required  the  greatest  number  of  modifications  and  was  renamed  the 
Research  and  Development  Impact  Analysis  (RDIA) .  Description  of 
the  DEFT,  the  general  MAUT  method,  and  the  RDIA  follow. 

DEVICE  EFFECTIVENESS  FORECASTING  TECHNIQUE  (DEFT) 

The  DEFT  was  developed  through  a  joint  effort  by  the  Army 
Research  Institute  and  the  American  Institutes  for  Research. 
The  following  summary  account  of  DEFT  is  based  on  the  detailed 
description  provided  by  Rose,  Wheaton,  and  Yates  (1985).  The 
DEFT  is  a  combination  of  four  major  analyses: 

1.  Analysis  of  the  training  problem 

2.  Analysis  of  (skill)  acquisition-efficiency 

3.  Analysis  of  the  transfer  problem 

4.  Analysis  of  transfer-efficiency. 
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Analysis  of  the  training  problem  is  examining  the  deficiency  in 
skills  and  knowledge  of  the  typical  trainee  relative  to  the 
trainee's  criterion  performance  on  the  training  device.  That  is, 
how  deficient  is  the  typical  trainee  in  the  desired  performance 
prior  to  being  introduced  to  the  training  device?  The  criterion 
performance  is  the  expected  performance  on  the  device,  not  the 
operational  equipment.  This  analysis  also  takes  into  account  a 
typical  trainee's  difficulty  in  learning  to  overcome  the  deficit. 

Aaalygjg  .af . JaKUrU _ saulsltlon-efflclencv  deals  with  the  quality 

of  training.  Which  instructional  features  and  training 
principles  does  the  device  have  which  will  help  the  trainee  to 
overcome  his/her  deficit? 

Analysis  of  the  transfer  problem  looks  at  the  remaining  deficit 
of  the  trainee  in  operational  criterion  performance.  That  is, 
once  the  trainee  has  completed  device  training,  what  is  the 
trainee's  remaining  deficiency?  The  functional  and  physical 
similarity  between  the  training  device  and  the  operational 
equipment  is  also  analyzed. 

Analysis  of  transfer-efficiency  investigates  how  well  the  device 
promotes  transfer  of  learning  to  the  actual  equipment. 

The  total  effectiveness  score  is  a  composite  of  these  four 
analyses.  Figure  2  illustrates  how  the  DEFT  score  is  computed. 
The  (skill)  acquisition  index  is  derived  from  the  first  and 
second  analyses;  i.e.,  analysis  of  the  training  problem  and 
analysis  of  (skill)  acquisition-efficiency;  and  transfer  is 
derived  from  the  third  and  fourth  analyses,  i.e.,  the  analysis  of 
the  transfer  problem  and  the  analysis  of  transfer-efficiency. 
Since  the  magnitude  of  skill  acquisition  depends  on  the  magnitude 
of  the  training  problem  and  how  efficiently  the  device  teaches 
the  trainee,  the  training  problem  index  is  divided  by  the  (skill) 
acquisition-efficiency  index;  the  transfer  index  is  similarly 
computed  by  dividing  the  transfer  problem  index  by  the 
transfer-efficiency  index.  In  the  end,  the  DEFT  provides  a  total 
score  which  is  a  numerical  estimate  of  device  effectiveness. 
Additionally,  by  looking  at  the  composite  scores,  diagnostic 
information  of  potential  strengths  and  weaknesses  may  be 
examined.  For  example,  a  high  total  score  could  be  the  result  of 
any  one  of  the  four  analyses  being  high  or  a  combination  of  any 
of  these  analyses  being  high.  The  authors  question  the 
usefulness  of  a  single  composite  score  for  other  than  comparing 
one  alternative  with  another.  More  useful  effectiveness 
infomnation  is  contained  in  the  acquisition  and  transfer  analyses 
scores ,  themselves . 
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TOTAL  EFFECTIVENESS  =  (SKILL)  ACQUISITION  ^  TRANSFER 


(SKILL)  ACQUISITION  =  TRAINING  PROBLEM  /  ACQUISITION  EFFICIENCY 
TRANSFER  =  TRANSFER  PROBLEM  /  TRANSFER  EFFICIENCY 
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The  DEFT  embraces  a  variety  of  siibjects,  skills,  and 
dimensions.  This  versatility  makes  this  method  ideal  for 
applying  to  R&D  projects/programs.  However,  the  quality  of  DEFT 
data  depends  on  the  detail  of  the  available  information  and  the 
diagnostic  ability  of  the  analyst.  Modifications  of  the  DEFT  as 
applied  to  the  selected  programs  are  presented  in  Appendix  A. 

MDLTZ-ATTRZBDTE  DTZLZTY  (MAUT)  METHOD 

The  MAUT  method  assesses  relative  preferences  for  possible 
outcomes  of  a  decision  that  can  be  represented  on  several 
dimensions  (Hogarth,  1987).  Hogarth  (1987)  subdivides  the 
approximately  ten  steps  within  the  MAUT  into  four  subgroups: 

o  Structuring  the  Problem 

o  Determining  the  Importance  of  Dimensions 

o  Measuring  Alternatives  on  the  Dimensions 

o  Making  a  Choice. 

Btrueturina  the  Problem  consists  of:  1)  identifying  the  decision 
maker (s) ;  2)  identifying  the  decision  (what?  and  why?) ;  3) 

identifying  the  alternatives;  and  4)  identifying  the  dimensions 
on  which  the  alternatives  will  be  evaluated. 

Determining  the  Zmoortanee  of  the  Dimensions  consists  of:  5) 
rank-ordering  of  the  dimensions  in  terms  of  importance;  6) 
translating  the  rankings  into  ratings  (basically  assigning 
weights  to  the  dimensions) ;  and  7)  converting  ratings  to 
numbers . 

Measuring  the  Alternatives  is  simply;  8)  evaluating  each 
alternative  across  all  dimensions;  and  9)  calculating  each 
alternative's  score  as  weighted  by  the  appropriate  importance 
weights. 

Making  a  Choice  is;  10)  Choosing  the  alternative  with  the 
largest  assessment  of  worth. 

Choices  based  on  worth  do  not  necessarily  indicate  that  the 
attributes  were  ranked  and  rated  according  to  cost  factors. 
Although  importance  weights  can  include  costs,  usually  costs  are 
determined  separately  and  predictions/evaluations  are  based  on  a 
combination  of  sources.  This  combination  of  information  sources 
can  best  be  explained  by  tne  following  example. 

Table  2  shows  how  a  MAUT  method  would  compute  scores  for 
five  alternatives:  interactive  videodisc  (VIDEOD) ,  instructor- 
led  group  discussions  (INSTR) ,  videotape  (VIDEOT) ,  exercises  in 
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Table  2 

Evaluation  of  Media  Alternatives 
for  Instructional  Problems 

DIMENSIONS 

WGT 

VIDEOD 

INSTR  VIDEOT 

HBK 

35MM 

Safety 

3 

4 

5 

3 

2 

1 

Motivation 

3 

4 

5 

3 

1 

2 

Conmunication 

3 

5 

2 

4 

1 

3 

Practice 

3 

5 

3 

2 

4 

1 

Standardization 

3 

5 

3 

2 

4 

1 

Tine 

3 

5 

4 

3 

2 

1 

Info  Disen 

2 

4 

5 

3 

2 

1 

Feedback 

2 

4 

5 

2 

3 

1 

Update 

2 

3 

5 

2 

4 

1 

Renediation 

2 

5 

4 

3 

2 

1 

Flexibility 

2 

4 

5 

2 

3 

1 

Human  Interact 

2 

3 

5 

4 

2 

1 

Leadership 

2 

4 

5 

3 

2 

1 

• 

Sensitivity 

1 

4 

5 

3 

2 

1 

Transition 

1 

4 

5 

2 

3 

1 

Pacing 

1 

5 

2 

4 

3 

1 

Testing 

1 

5 

4 

2 

3 

1 

Scope  of  Tasks 

1 

5 

4 

3 

2 

1 

Scoring 

1 

5 

3 

2 

4 

1 

Total 

166 

157 

105 

95 

47 
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a  workbook  (WBK) ,  and  35inia  slides  (35MM)  .  The  data  represented 
in  the  table  were  taken  from  an  evaluation  conducted  on  the 
Interactive  Videodisc,  Device  11H89  (Pfeiffer,  Miller,  Platt, 
Green,  Monroe,  &  Traxler,  1986) .  Some  of  these  data  have  been 
transformed  to  aid  in  this  illustration.  By  examining  the  table, 
one  can  see  that  the  nineteen  dimensions  had  differing  levels 
of  importance;  consequently,  the  dimensions  were  grouped  and 
weighted  accordingly. 

Based  on  these  data,  the  interactive  videodisc  had  the 
highest  score  and  would  be  the  normative  choice.  However,  in 
light  of  cost  estimates  one  may  have  to  reconsider.  Cost  data 
were  collected  and  are  represented  in  Figure  3.  Figure  3  depicts 
the  present  value  costs  against  the  training  effectiveness 
rankings  as  computed  by  the  MAUT  method.  Because  of  the  much 
lower  cost  ($0.5  million)  of  the  instructor-led  group  discussion 
versus  the  high  cost  ($2.5  million)  of  the  interactive  videodisc, 
the  original  normative  choice  of  the  interactive  videodisc  does 
not  appear  to  be  the  correct  one.  In  fact,  considering  the 
Decision  Rules  for  Choosing  Alternatives,  Table  1  (p.  14),  this 
example  meets  the  third  case  of  different  life  cycle  cost  and 
different  training  effectiveness;  the  rule  states  to  select  the 
higher  training  effectiveness  and  lower  life  cycle  cost  or  use  a 
subjective  appraisal  of  the  estimates  with  no  specific  rule  to 
apply.  Since  neither  alternative  meets  the  first  half  of  the 
rule,  then  the  decision  has  to  be  made  subjectively.  By 
examining  the  differences  of  the  estimates,  it  can  be  seen  that 
the  greatest  difference  is  in  the  life  cycle  cost  estimates. 
Therefore,  in  this  analysis,  the  instructor-led  group  discussion 
should  be  chosen  over  the  interactive  videodisc. 

In  summary,  the  MAUT  method  combined  with  cost  estimates 
produces  a  defensible  rationale  for  decision  making.  This 
methodology  provided  the  basis  for  developing  the  Research  and 
Development  Impact  Analysis  (RDIA) .  The  following  is  a 
description  of  the  RDIA  with  an  explanation  of  the  changes  which 
were  made  to  the  MAUT  method. 

RESEARCH  AND  DEVELOPMENT  IMPACT  ANALYSIS  (RDIA) 

The  Research  and  Development  Impact  Analysis  is  a  variation 
of  the  Multi-Attribute  Utility  method  and  was  a  better  fit  for 
the  four  specific  R&D  projects.  The  RDIA  bypassed  some  of  the 
steps  of  the  MAUT  method.  The  alterations  included: 

o  General  dimensions  which  may  be  appropriate  for  most 
programs  were  chosen  over  specific  dimensions. 
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COST  AND  EFFECTIVENESS 


35  MMS  WORKBK  VIDEOT  IN8TR  VIDEOD 

TRAINING  EFFECTIVENESS 


RANKING  LOW  TO  HIGH 


Figure  3.  Present  Value  Costs  vs  Training  Effectiveness 
Rankings. 
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o  The  rating  scales  were  designed  to  compare  the  proposed 
training  R&D  to  a  common  alternative  (i.e.,  the  current 
training  procedures)  instead  of  comparing  multiple 
alternatives . 

o  The  dimensions  chosen  had  equal  ranks  and  weights; 
importance  weights  can  be  added  if  deemed  necessary. 

The  RDIA  produces  an  overall  score  representing  the  difference 
between  the  R&D  project/program  and  an  existing  alternative.  The 
higher  the  score,  the  better  the  program  is  over  that  which  is 
already  in  place.  Appendix  B  contains  copies  of  the  ROIA  as 
applied  to  the  selected  R&D  programs. 

COSTING  APPROACH 

This  approach  to  costing  is  tailored  to  R&D  projects.  There 
is  no  comprehensive  costing  model,  such  as  Knapp  &  Orlansky 
(1985),  because  at  this  stage  in  system/device  development  many 
resource  requirements  are  unknown.  Table  3  lists  the  steps 
necessary  to  do  a  basic  cost  analysis.  This  is  a  summary  of  the 
process  described  in  the  second  edition  of  the  DOD  Economic 
Analysis  Handbook. 

The  first  task  is  a  statement  of  the  problem;  where  will 
this  R&D  product  fit  in  the  current  training  pipeline?  Is  it  a 
substitute  for  something  being  done  now  or  is  it  an 
enhancement/upgrade  to  an  existing  trainer/device?  Before 
anything  else  can  be  started  in  the  analysis,  there  must  be  a 
clear  picture  of  the  training  requirements  that  the  training  R&D 
product  will  satisfy. 

The  second  step  is  to  identify  alternatives  that  can  satisfy 
these  training  requirements;  there  may  be  many  or  there  may  be 
none.  This  is  the  hardest  and  most  important  part  of  the 
analysis.  Some  alternatives  may  be  dismissed  because  of 
constraints  on  labor,  facilities,  or  funding. 

Next,  the  resources  needed  to  implement  each  alternative 
must  be  identified.  Resources  for  acquisition  and 
implementation,  as  well  as  resources  used  for  R&D,  should  be 
included.  The  word  "resource”  includes  manpower,  equipment, 
supplies,  facilities,  and  utilities.  These  resources  are  used  in 
everything  from  maintenance  to  curriculum. 
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Table  3 

Tasks  Necessary  to  Perform  a  Cost  Analysis 

1.  STATEMENT  OF  THE  PROBLEM 

2 .  IDENTIFY  ALTERNATIVES 

3.  IDENTIFY  RESOURCES  REQUIRED  FOR  EACH  ALTERNATIVE 

4.  DETERMINE  COST  OF  THE  RESOURCES 

5.  DETERMINE  TOTAL  COST  OF  EACH  ALTERNATIVE 

6.  TIME  PHASE-COMPUTE  PRESENT  VALUE 

7.  COMPARE  PRESENT  VALUE  COST  OF  ALTERNATIVES 


The  resources  identified  for  each  alternative  must  then  be 
costed.  The  per  unit  price  of  each  ''resource”  and  the  amount 
used  need  to  be  estimated.  The  quantity  consumed  of  a  resource 
will  depend  on  the  time  span;  most  of  these  analyses  cover  10-15 
years.  Simplifying  assumptions  include  (1)  year  1  is  all  R&D 
expenditures;  (2)  year  2  is  implementation;  (3)  steady  state  in 
all  other  years;  and  (4)  inflation  is  assumed  to  be  identical  for 
all  resource  categories  and  therefore  is  excluded  from  the 
analysis. 

To  calculate  the  total  costs  of  each  alternative,  step  5,  a 
spreadsheet  should  be  set  up.  Table  4  is  a  generic  spreadsheet; 
it  covers  a  ten-year  period  with  major  cost  categories.  For 
specific  projects,  of  course,  the  time  frame  and  the  cost 
categories  could  be  different.  Student  salaries  are  included  and 
are  defined  as  the  value  of  the  student  time  spent  in  training  on 
a  particular  alternative. 

To  calculate  this  value,  it  is  necessary  to  know  the  time 
spent  by  the  average  student,  the  number  of  students,  and  the 
salary  of  the  average  student.  The  other  cost  categories  should 
be  self-explanatory.  Not  all  cost  categories  will  have  entries 
each  year,  such  as  training  hardware  which  would  be  a  one-time 
acquisition  cost. 
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Table  4 

Generic  Cost  Analysis  Spreadsheet 


Yefr 

Training 
Hardware  Fae 

:i  titles 

mi 

wmm 

student 

Salaries 

Total 

Cost 

Discounted 

Cost 

1 

$0 

SO 

SO 

SO 

so 

SO 

SO 

SO 

2 

SO 

SO 

SO 

so 

so 

SO 

SO 

SO 

3 

so 

SO 

SO 

so 

so 

so 

so 

SO 

4 

SO 

so 

SO 

so 

so 

so 

so 

SO 

5 

so 

SO 

SO 

so 

so 

so 

so 

SO 

6 

so 

so 

so 

so 

so 

so 

so 

SO 

7 

so 

$0 

so 

so 

so 

so 

so 

SO 

8 

so 

so 

so 

so 

so 

so 

so 

so 

9 

SO 

so 

so 

so 

so 

so 

so 

A 

10 

so 

so 

so 

so 

so 

so 

so 

Total 

SO 

so 

so 

so 

so 

so 

so 

so 
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Table  4  has  two  total  cost  col\mns;  the  first  is  a  straight 
siunmation  across  the  rows,  while  the  second  is  a  discounted 
total  cost,  or  a  present  value  cost  (step  6) .  This  step  is  added 
because  alternatives  frequently  have  different  expenditure 
patterns  over  time.  For  example,  one  project  may  spend  most  of 
its  resources  in  the  future,  while  an  alternative  may  spend  most 
of  its  resources  up  front.  To  calculate  the  discounted  total 
cost  for  any  given  year,  the  undiscounted  total  cost  (the 
preceding  coltimn)  is  multiplied  by  a  discount  factor;  this  factor 
reflects  the  change  in  the  value  of  a  dollar  over  time.  Table  5 
lists  the  discount  factors  for  any  analysis  up  to  25  years.  If, 
in  the  sixth  year  of  a  project,  expenditures  are  going  to  total 
$10,000,000,  then  the  discounted  cost  would  be  $5,920,000 
($10,000,000  X  0.592) . 

The  final  step  in  the  cost  analysis  is  simply  to  compare  the 
summation  of  the  discounted  costs  of  each  alternative.  For 
planning  purposes,  this  cost  is  the  better  indicator  of  resource 
expenditures  over  time. 

The  factors  are  based  on  continuous  compounding  of  interest 
assuming  uniform  cash  flows  throughout  the  one-year  period. 
These  factors  are  equivalent  to  an  arithmetic  average  of 
beginning  and  end  of  year  compound  amount  factors  found  in 
standard  present  value  tables.  Ten  percent  is  used  in  this 
example  since  it  is  the  DoD-established  discount  rate  (DoDI 
7041.3).  The  next  section  provides  sample  applications  of  this 
cost  approach  along  with  training  effectiveness  measures  for  the 
four  R&D  programs. 
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Table  5 

Discount  Factor 


PROJECT  PRESENT  VALUE 

YEAR  OF  $1 


1 

0.954 

2 

0.867 

3 

0.788 

4 

0.717 

5 

0.652 

6 

0.592 

7 

0.538 

8 

0.489 

9 

0.445 

10 

0.405 

11 

0.368 

12 

0.334 

13 

0.304 

14 

0.276 

15 

0.251 

16 

0.228 

17 

0.208 

18 

0.189 

19 

0.172 

20 

0.156 

21 

0.142 

22 

0.129 

23 

0.117 

24 

0.107 

25 

0.097 
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SAMPLE  APPLZCATZOMS  OF  ZMPACT  ANALYSES 

Approximately  150  R&D  projects/programs  are  in  progress  at 
the  Naval  Training  Systems  Center.  These  efforts  differ  on  many 
factors  such  as  purpose,  scope,  cost,  quantifiability  of  results, 
etc.  Code  7  chose  for  Impact  analyses  the  following  projects/ 
programs  as  a  cross  representation  of  their  many  programs. 

RtD  PROGRAM  DESCRZPTZOMS 


This  project  is  developing,  demonstrating,  and  evaluating 
embedded  training  capability  in  the  AN/SPA-2 5G  radar 
repeater.  The  focus  of  this  capability  is  to  provide  skill 
training  in  radar  fundamentals  for  OS  "A"  School  graduates, 
"pre-training”  for  AIC  "C"  School  candidates,  and  refresher 
training  for  AIC  "C"  School  graduates. 


T  ^*1  1  f  !  i  r  1  1  W~l  7  I  M  I  1  I  'll  M'il 


The  TRIO  system  trains  undergraduate  naval  flight 
officers  in  basic  air  intercept  tactics.  An  important  aspect  of 
TRIO  is  its  visual  and  operational  simulation  which  incorporates 
sophisticated  instructional  facilities.  For  example,  TRIO 
provides  examples  to  the  student  using  an  expert  system.  The 
expert  system  performs  intercepts  and  explains  its  actions  and 
strategy  to  the  student  RIO.  Also,  TRIO  generates  performance 
analysis  summaries  containing  specific  error  diagnostics  after 
student  practice  runs. 


\  i  l  .  W-Plri#  1 


'jfTTsTji 


The  PAA  trainer  uses  high  resolution  displays  and 
synchronized  audio  simulation  to  train  sonar  operators.  Tactical 
tapes  are  processed  at  a  software  support  facility  to  provide  the 
same  sonar  information  that  a  tactical  operator  would  see  and 
hear.  This  technique  produces  an  exact  replicate  of  the  tactical 
environment  represented  by  LOFAR  grams.  Realistic  displays  are 
generated  by  digitally  duplicating  the  display  that  is  obtained 
by  playing  a  taped  acoustic  contact  into  a  sonar  set. 


The  MTS,  sometimes  referred  to  as  the  Markmanship  Expert 
Trainer  (MET) ,  instructs  rifle  marksmanship  to  navy  recruits 
without  an  instructor,  real  weapon,  or  rifle  range.  The  system 
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is  controlled  by  a  Zenith  248  personal  computer.  The  MTS 
consists  of  six  major  parts:  (a)  long  range  light  pen,  (b) 
Zenith  248,  (c)  color  monitor,  (d)  computer  speech  board,  (e) 
analog  and  digital  board,  and  (f)  force  sensing  resistors. 

A  long  range  light  pen  is  attached  to  an  M-14  rifle  and 
targets  are  displayed  on  the  Zenith  248  monitor.  The  light  pen 
is  used  to  determine  hits  on  the  target  and  tracking  steadiness. 
A  breath  sensor  is  placed  around  the  trainee's  diaphragm  to 
detezmine  if  he  held  his  breath  while  firing  the  weapon.  A  force 
sensing  resistor  is  used  on  the  trigger  to  determine  how  the 
trainee  squeezed  the  trigger  when  firing  the  weapon.  Feedback  is 
provided  by  computer  generated  voice  and  monitor  graphics.  Bang 
and  recoil  of  the  weapon  are  also  simulated. 

SAMPLE  APPLICATIONS 

Approximately  four  to  five  Subject  Matter  Experts 
(SMEs)  identified  by  the  principal  investigators  of  each  of  the 
R&D  programs  were  administered  the  DEFT  and  the  RDIA.  Also  a 
standard  life-cycle  cost  analysis  was  performed  to  determine  cost 
estimates.  The  remainder  of  this  section  will  describe  the 
outcome  of  applying  this  approach  to  the  programs.  Each  program 
is  treated  separately  and  is  presented  in  the  following  order; 
first,  the  DEFT  results,  then  the  results  of  the  RDIA,  and, 
finally,  the  cost  determination  estimates. 

Embedded  Training  Analysis 

DEFT.  Each  SME  was  instructed  to  rate  a  typical  trainee  on 
three  dimensions  within  embedded  training;  i.e.,  (1)  Visual 
Display,  (2)  Manual  Control,  and  (3)  Communication.  Addition¬ 
ally,  each  SME  was  to  use  three  frames  of  reference  while  rating 
the  dimensions.  These  frames  of  reference  were  a  typical  trainee 
who  had  graduated  from  A  School,  a  typical  trainee  who  was  a 
potential  Air  Intercept  Control  (AIC)  student,  and  a  typical 
trainee  who  was  an  Air  Intercept  Control  graduate.  Table  6 
presents  the  composite  scores  and  the  total  effectiveness  score 
for  the  three  dimensions  and  frames  of  reference.  The  DEFT 
scores  were  low  suggesting  that  this  embedded  training  system 
will  be  effective.  Noteworthy  is  that  the  Communication 
Dimension  had  higher  scores  across  all  frames  of  reference.  Of 
the  three  dimensions  chosen,  communication  was  least  appropriate 
and,  as  expected,  produced  higher  scores.  Another  area  in  which 
the  DEFT  was  sensitive  was  for  the  pre-AIC  graduate.  Pre-AIC 
graduates  learn  rudimentary  functions  in  A  School  and  then  are 
exposed  to  radar  systems  in  the  fleet;  they  are  not  usually 
familiar  with  the  more  complex  tasks  of  the  radar.  The  higher 
DEFT  scores  for  the  pre-AIC  graduates  reflect  this  phenomenon. 
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Table  6 

DEFT  Scores  for  Embedded  Training 


Dimension  A  School _ Pre  AIC _ Post  AIC 


Training  Problem 

Visual  Display 

5.05 

20.63 

0.86 

Manual  Control 

1.20 

6.64 

0.14 

Communication 

36.75 

31.50 

2.41 

(Skill) 

Acquisition  -  Efficiency 

Visual  Display 

0.82 

0.86 

0.92 

Manual  Control 

0.84 

0.80 

0.82 

Communication 

0.57 

0.67 

0.74 

(Skill)  Acquisition 


Visual  Display 
Manual  Control 
Communication 

6.16 

1.43 

64.47 

23.99 

8.30 

47.01 

0.93 

0.17 

3.26 

Transfer  Problem 

Visual  Display 

35.00 

50.00 

15.00 

Manual  Control 

37.50 

47.5 

13.75 

Communication 

56.67 

52.50 

26.25 

Transfer  • 

-  Efficiency 

Visual  Display 

0.99 

0.99 

0.99 

Manual  Control 

1.00 

1.00 

1.00 

Communication 

0.77 

0.91 

0.98 

Transfer 


Visual  Display 

35.35 

50.51 

15.15 

Manual  Control 

37.50 

47.50 

13.75 

Communication 

73.60 

57.69 

26.79 

Total 

Effectiveness 

Visual  Display 

41.51 

74.50 

16.08 

Manual  Control 

38.93 

55.80 

13.75 

Communication 

138.07 

104.70 

30.05 
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RDIA.  The  ROIA  was  given  to  the  same  SMEs  who  filled  out 
the  DEFT.  Table  7  presents  the  ratings  and  total  scores  obtained 
from  these  SMEs.  Converse  to  the  DEFT,  a  higher  score  on  the 
RDIA  suggests  a  more  effective  training  system.  Out  of  a 
possible  45,  each  SHE  rated  this  system  34  or  higher  (the 
average  total  score  was  36.5),  again,  suggesting  an  effective 
training  system. 

cost  Estimates .  Table  8  shows  the  life  cycle  costs  (LCC) 
associated  with  implementation  of  embedded  training  for  AIC 
personnel.  Simplifying  assumptions  are  that  development  took 
place  in  Year  1  and  equipment  was  installed  in  Year  2.  Also, 
it  was  assumed  that  300  ships  would  have  this  embedded  training 
capability  with  each  having  one  target  generator  and  an  average 
of  seven  Zenith  248  personal  computers.  This  equipment  is  90%  of 
the  $11.5  million  LCC;  the  discounted  (10%  discount  factor)  LCC 
is  $9.5  million.  Personnel  requirements  were  assumed  to  be 
unchanged  from  current  training  and  therefore  are  not  included. 

Trainer  for  Radar  Intercept  Operator  Analysis  (TRIO^ 

DEFT.  Three  dimensions  of  the  TRIO  were  rated.  These 
dimensions  were  (1)  Visual  Display,  (2)  Audio  Present,  and  (3) 
Oral  Communication.  Table  9  presents  the  results  of  applying 
the  DEFT  to  the  TRIO.  The  total  effectiveness  score  and  its 
composite  scores  were  all  low,  thus,  indicating  an  effective 
training  system.  Of  the  three  dimensions,  the  Visual  Display 
dimension  produced  the  highest  (worst)  score. 

RDIA.  Table  10  presents  the  ratings  and  total  scores  for 
The  TRIO  as  obtained  from  four  SMEs.  The  average  total  score  was 
40;  "availability  to  practice"  and  "feedback  to  the  trainer" 
received  the  highest  rating.  According  to  the  RDIA,  the  TRIO  is 
an  effective  training  system. 

Cost  Estimates.  Table  11  shows  the  LCC  associated  with 
implementing  TRIO  into  the  training  of  F-14  radar  intercept 
officers.  Development  and  acquisition  of  TRIO  is  assumed  to 
occur  in  the  first  two  years  of  the  10  year  life  cycle.  It  is 
also  assumed  that  the  hardware  for  one  TRIO  unit  is  $35K;  a  total 
of  34  units  would  be  purchased  for  the  squadrons,  reserve  units, 
and  Replacement  Air  Group  Squadron.  This  equipment  and  the 
development  costs  of  $1,285M  would  constitute  about  85%  of  the 
LCC  of  $2,935M;  the  discounted  LCC  is  $2.4M.  Personnel 
requirements  were  assumed  to  be  unchanged  from  current  training 
and  therefore  not  included. 
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Table  7 


Research  &  Development  Impact  Analysis 
for  Embedded  Training 


Questions 

#1 

Raters 
#2  #3 

«4 

Mean 

Training 

Time 

5 

4 

4 

4 

4.25 

Required 

Instructor  Time 

5 

5 

4 

5 

4.75 

Training 

Content 

2 

4 

4 

4 

3.50 

User 

Acceptance 

4 

4 

4 

5 

4.25 

Instructor 

Acceptance 

4 

3 

2 

5 

3.50 

Availability 
to  practice 

5 

5 

5 

5 

5.00 

Feedback  to 
trainee 

4 

4 

4 

5 

4.25 

Student 

Performance 

Speed 

3 

3 

3 

4 

3.25 

Student 

Performance 

Accuracy 

3 

4 

4 

4 

3.75 

Total 

35 

36 

34 

41 

36.50 

35 
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# 

Table  8 

Life  Cycle  Costs  for  Enbedded  Training 


ZtiL 

Training 

It  Hardware  f 

'acilities 

Maintenance  1 

nstnictors 

Student 

Salaries 

Total 

Cost 

Discounted 

9ost 

1 

8910,000 

80 

80 

80 

80 

80 

8910,000 

8827,273 

2 

80 

810,300,000 

80 

810,000 

80 

80  810,510,000 

88,685,950 

3 

80 

80 

80 

810,000 

80 

SO 

810,000 

87,513 

4 

80 

SO 

SO 

810,000 

SO 

SO 

810,000 

86,830 

5 

SO 

so 

SO 

810,000 

SO 

SO 

810,000 

86,209 

6 

80 

so 

SO 

810,000 

SO 

SO 

810,000 

85,645 

7 

80 

so 

SO 

810,000 

SO 

so 

810,000 

S5,132 

8 

SO 

so 

so 

810,000 

SO 

so 

810,000 

84,665 

9 

SO 

so 

so 

810,000 

SO 

so 

810,000 

84,241 

10 

SO 

so 

so 

810,000 

SO 

so 

810,000 

83,855 

8910,000 

810,500,000 

so 

890,000 

SO 

80  S11 

,500,000 

89,557,313 
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Table  9 

DEFT  Scores  for  the  TRIO 


Visual 

Audio 

Oral 

Training 

Problem 

22.55 

9.34 

24.94 

(Skill)  Acquisition  -  Efficiency 


0.94  0.95  0.95 


(Skill)  Acquisition 


23.99  9.83  26.25 


Transfer  Problem 


61.25  40.00  38.75 


Transfer  -  Efficiency 


0.89  0.93  0.97 


Transfer 


68.82  43.01  39.95 


Total  Effectiveness 


92.81  52.84  66.20 
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Table  10 


Research  &  Development  Impact  Analysis 
for  the  TRIO 


Questions 

#1 

#2 

Raters 

#3 

#4 

Mean 

Training 

Time 

3 

5 

4 

3 

3.75 

Reg'd  Instr. 

Time 

5 

5 

4 

3 

4.25 

Training 

Content 

4 

3 

3 

3 

3.25 

User 

Acceptance 

4 

5 

4 

5 

4.50 

Instructor 

Acceptance 

Availability 

5 

5 

4 

5 

4.75 

• 

to  Practice 

5 

5 

5 

5 

5.00 

Feedback  to 

Trainee 

Student 

Performance 

5 

5 

5 

5 

5.00 

Speed 

Student 

Performance 

4 

5 

5 

5 

4.75 

Accuracy 

4 

5 

5 

5 

4.75 

Total 

39 

43 

39 

39 

40.00 
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Training  Student  Total  Discounted 


[mwnrra 

B-'n* 

1 

SI, 285,000 

SO 

so 

so 

so 

so 

SI, 285,000 

SI, 168, 182 

2 

$0 

SI, 200,000 

so 

S50,000 

so 

so 

SI, 250,000 

SI, 033,058 

3 

SO 

SO 

so 

S50,000 

so 

so 

S50,000 

S37,513 

4 

SO 

SO 

so 

S50,000 

so 

so 

S50,000 

S34,151 

5 

$0 

so 

so 

S50,000 

so 

so 

S50,000 

S31,046 

6 

so 

so 

so 

S50,000 

so 

so 

S50,000 

S28,224 

7 

so 

so 

so 

S50,000 

so 

so 

S50,000 

S25,658 

8 

$0 

so 

so 

S50,000 

so 

so 

S50,000 

S23.325 

A 

so 

so 

so 

S50,000 

so 

so 

S50,000 

S2 1,205 

• 

so 

so 

so 

S50,000 

so 

so 

S50,000 

S19,277 

SI,  285, 000 

SI, 200, 000 

so 

S450,000 

so 

so 

S2, 935, 000 

S2, 421, 691 
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gattsive  Aooastic  Analysis  Trainer  (PAAT)  Analysis 

BEia.  The  PAAT  was  rated  on  four  dimensions:  (l)  Visual 
Display,  (2)  Audio  Present,  (3)  Manual  Control,  and  (4) 
Communication.  Table  12  displays  the  results  obtained  from  the 
SME  ratings.  The  Audio  Present  dimension  received  the  lowest 
(best)  score  of  the  four  dimensions.  However,  all  scores  were 
low  indicating  an  effective  training  system. 

RDIA.  This  analysis  also  suggested  an  effective  training 
system.  Table  13  presents  the  ratings  and  total  scores  from  each 
of  four  SMEs.  None  of  the  total  scores  was  below  35,  and  five  of 
the  nine  factors  had  a  rating  of  5.  The  only  poor  scoring  factor 
was  "training  time."  All  SMEs  marked  that  there  would  be  an 
increase  in  training  time  using  the  PAAR  versus  using  the  current 
training  system/procedures. 

Cost  Estimates .  Table  14  shows  the  LCC  of  developing, 
purchasing,  operating,  and  maintaining  nine  passive  acoustic 
analysis  trainers.  Development  takes  place  in  Year  1  and  amounts 
to  acquisition  of  hardware  and  facilities  preparation 
occur  in  Year  2  and  cost  $7.1M  and  $0.5M  respectively. 
Maintenance  of  the  devices  is  expected  to  be  $225K  annually.  LCC 
were  estimated  to  be  $10.7M;  the  discounted  LCC  were  $8.5M. 
Personnel  requirements  were  assumed  not  to  be  higher  than  those 
used  for  current  training  and  therefore  were  not  included. 

An  alternative  to  using  the  PAAT  would  be  the  tactical 
equipment,  the  BQQ-5.  The  acquisition  cost  of  one  BQQ-5  is  about 
$25M.  If  nine  BQQ-5s  could  train  as  many  students  as  nine  PAATs 
(it  would  actually  take  more  BQQ-5s) ,  this  option  would  cost 
approximately  $300M  for  just  the  major  hardware  purchase,  initial 
spares,  and  site  preparation.  Because  of  the  huge  difference  in 
the  hardware  costs  between  the  PAAT  and  the  BQQ-5,  a  detailed  LCC 
of  the  BQQ-5  alternative  is  not  presented.  These  cost  data  are 
conclusive  evidence  that  the  PAAT  is  a  less  expensive  option  than 
the  tactical  equipment. 

Marksmanship  Training  Simulator  Analysis 

DEFT.  The  MTS  was  rated  on  three  dimensions.  These 
dimensions  were  (1)  Visual  Display,  (2)  Audio  Present,  and  (3) 
Manual  Control.  Table  15  displays  the  DEFT  scores  as  computed 
from  the  ratings  of  the  SMEs.  The  total  effectiveness  score  and 
composite  scores  for  all  three  dimensions  were  low  and, 
therefore,  indicated  that  the  MTS  is  an  effective  training 
system.  The  Audio  Present  dimension  had  the  lowest  (best)  score, 
while  the  Manual  Control  dimension  had  the  highest  (worst)  score. 
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Table  12 

DEFT  Scores  for  the  PAA  Trainer 

i 


visual 

Audio 

Manual 

Communication 

Display 

Present 

Control 

Training  Problem 

2.88 

3.36 

23.50 

15.12 

(Skill) 

Acquisition  -  Efficiency 

0.93 

0.91 

0.94 

0.78 

(Skill)  Acquisition 

Tm 

3.10 

3.69 

25.00 

19.38 

Transfer  Problem 

59.0 

5.75 

62.75 

54.25 

Transfer  -  Efficiency 

0.80 

0.90 

0.61 

0.77 

Transfer 

73.75 

6.39 

102.87 

70.45 

Total  Effectiveness 

• 

76.85 

10.08 

127.87 

89.83 
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Table  13 

Research  &  Development  Impact  Analysis 
for  the  PAA  Trainer 

• 

Questions 

Raters 

Mean 

#1 

#2 

#3 

«4 

Training 

Tine 

1 

1 

1 

1 

1.00 

Req'd  Instr. 

Time 

4 

4 

5 

5 

4.50 

Training 

Content 

5 

5 

5 

4 

4.75 

User 

Acceptance 

5 

5 

5 

5 

5.00 

Instructor 

Acceptance 

5 

5 

5 

5 

5.00 

Availability 

# 

to  Practice 

5 

5 

5 

5 

5.00 

Feedback  to 

Trainee 

5 

3.5 

5 

4 

4.40 

Student 

Performance 

Speed 

5 

5 

5 

5 

5.00 

Student 

Performance 

Accuracy 

5 

5 

5 

5 

5.00 

Total 

40 

38.5 

41 

39 

39.6 
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Table  14 


Life  Cycle  Costs  for  the  PAA  Trainer 


Training 

Harduare 

site  Prep 
Facilities 

IHIIil 

student 

Salaries 

Total 

Cost 

Discounted 

Cost 

1 

81,100,000 

SO 

SO 

so 

so 

SO  SI 

1,100,000 

81,100,000 

2 

SO 

87,100,000 

8500,000 

8225,000 

so 

SO  87,825,000 

86,486,942 

3 

so 

SO 

SO 

8225,000 

so 

SO 

8225,000 

8169,046 

4 

so 

SO 

SO 

8225,000 

so 

SO 

8225,000 

8153,678 

5 

SO 

SO 

so 

S22S,000 

so 

SO 

8225,000 

$139,707 

6 

so 

SO 

so 

8225,000 

so 

so 

S225,000 

8127,007 

7 

so 

so 

so 

8225,000 

so 

so 

8225,000 

S115,461 

8 

so 

$0 

$0 

8225,000 

so 

so 

8225,000 

S104,964 

9 

so 

so 

so 

$225,000 

so 

so 

8225,000 

895,422 

10 

so 

so 

so 

8225,000 

so 

so 

8225,000 

886,747 

81,100,000 

$7,100,000 

8500,000 

82,025,000 

so 

so  810,725,000 

88,458,974 
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Table  15 

88-017 

i 

DEFT  Scores  for  the  MTS 

Visual 

Display 

Audio 

Present 

Manual 

Control 

Training  Problem 

0.50 

0.06 

4.72 

(Skill)  Acquisition 

-  Efficiency 

0.96 

0.89 

0.91 

(Skill)  Acquisition 

0,52 

0.07 

5.19 

i 

1 

Transfer  Problem 

14.20 

5.40 

32.40 

Transfer  -  Efficiency 

0.83 

0.85 

0.91 

- 

Transfer 

17.11 

6.35 

35.60 

Total  Effectiveness 

17.63 

6.42 

40.79 

i 

1 
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Bm.  six  SMEs  rated  the  MTS  using  this  instrument.  Table 
16  presents  the  ratings  and  total  scores  obtained  from  these  six 
experts.  The  total  score  for  the  MTS  across  all  the  SMEs  was  at 
least  34  or  above  (the  average  score  was  36.70) ;  the  "student 
performance:  speed  and  accuracy"  factors  received  the  highest 
average  ratings  (4.5).  Over  all,  the  RDIA  suggests  that  the  MTS 
is  an  effective  training  system. 


Cost  Estimates .  Table  17  shows  the  LCC  of  the  MTS  for  a 
ten-year  period.  Development  costs  are  low  because  this  is  an 
adaptation  of  existing  technology.  The  initial  cost  of 
simulators,  $2.7M,  and  facilities,  $3.1M,  are  relatively  high. 
However,  the  biggest  cost  driver  is  personnel  expenditures;  this 
includes  both  additional  instructors  for  RTC  and  student  time  in 
training.  The  LCC  of  this  option  is  $90M;  this  is  equivalent  to 
a  discounted  LCC  of  $54M. 


The  LCC  for  the  option  using  M-14s  is  shown  in  Table  18. 
Acquisition  costs  in  Year  1  would  include  ranges,  weapons,  and 
troop  transport  (to  the  ranges) .  The  major  recurring  costs  would 
be  for  ammunition  and  personnel  (instructors  and  training  time) . 
The  LCC  for  this  option  is  $14 IM  or  over  50%  more  than  the  MTS 
alternative  (this  percent  also  holds  for  the  discounted  LCC) . 


Both  the  DEFT  and 
programs  were  training 
additional  information 
summarizes  the  analyses 
an  impact  analysis. 


the  RDIA  indicated  that  all  four  R&D 
effective.  Cost  estimates  provided 
for  decision  making.  The  next  section 
and  offers  guidelines  for  administering 
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Table  16 

88-017 

• 

Research  & 

Development 
for  the  MTS 

Impact  Analysis 

Questions 

#1 

#2 

Raters 
#3  #4 

#5 

#6 

Mean 

Training 

Time 

4 

4 

3.2 

4 

2 

5 

3.70 

Req'd  Instr. 
Time 

4 

4 

5 

5 

2 

3 

3.80 

Training 

Content 

2 

4 

5 

3 

4 

5 

3.80 

User 

Acceptance 

4 

3 

5 

4 

4 

5 

4.20 

Instructor 

Acceptance 

3 

3 

5 

3.3 

4 

5 

3.90 

Availability 
to  Practice 

4 

4 

4 

4 

5 

3 

4.00 

• 

Feedback  to 
Trainee 

3 

4 

5 

5 

4 

5 

4 . 30 

Student 

Performance 

Speed 

5 

4 

4 

4 

5 

5 

4.50 

Student 

Performance 

Accuracy 

5 

4 

4 

4 

5 

5 

4.50 

Total 

34 

34 

40.2 

36.3 

35 

41 

36.70 
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Table  17 

Life  Cycle  Costs  for  the  NarksaMnship  Training  Sisulator 


Year 

Training 
Pevelocaient  Hardware 

Site  Prep 
Facilities 

iHil 

Student 

Salaries 

Total 

Cost 

Discounted 

Cost 

1 

S150,000 

so 

SO 

so 

so 

SO 

$150,000 

$136,364 

2 

SO 

S2,669.000 

S3, 100,000 

sioo.ooo 

$2,500,000 

$6,750,000 

$15,119,000 

$12,495,041 

3 

SO 

SO 

SO 

SI 00, 000 

$2,500, 000 

$6,750,000 

$9,350,000 

$7,024,793 

4 

SO 

so 

$0 

SI 00. 000 

$2,500,000 

$6,750,000 

$9,350,000 

$6,386,176 

5 

so 

so 

so 

$100,000 

$2,500,000 

$6,750,000 

$9,350,000 

$5,805,614 

6 

so 

so 

so 

$100,000 

$2,500,000 

$6,750,000 

$225,000 

$5,277,831 

7 

so 

so 

so 

$100,000 

$2,500,000 

$6,750,000 

$225,000 

$4,798,028 

8 

so 

so 

so 

$100,000 

$2,500,000 

$6,750,000 

$225,000 

$4,361,844 

9 

so 

so 

so 

$100,000 

$2,500,000 

$6,750,000 

$225,000 

$3,965,313 

10 

so 

so 

so 

$100,000 

$2,500,000 

$6,750,000 

$225,000 

$3,604,830 

S1S0.000 

$2,669,000 

$3,100,000 

$900,000 

$22,500,000  $60,750,000 

$90,069,000 

$53,855,835 
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Table  18 

Life  Cycle  Costs  for  the  M'14  Alternative 


Tear 

UeaDons 

Ammo 

mm 

Troop 

Transoort 

Student 

Salaries 

Total 

Cost 

Discounted 

Cost 

1 

$138,000 

S1,23S,000 

$1,000,000 

$600,000 

$0 

$0 

$2,973,000 

$2,702,727 

2 

$0 

$1,235,000 

$50,000 

$60,000 

$5,000,000 

$9,000,000 

$15,345,000 

$12,681,818 

3 

$0 

$1,235,000 

$50,000 

$60,000 

$5,000,000 

$9,000,000 

$15,345,000 

$11,528,926 

4 

$0 

$1,235,000 

$50,000 

$60,000 

$5,000,000 

$9,000,000 

$15,345,000 

$10,480,841 

5 

so 

$1,235,000 

$50,000 

$60,000 

$5,000,000 

$9,000,000 

$15,345,000 

$9,528,038 

o 

so 

$1,235,000 

$50,000 

$60,000 

$5,000,000 

$9,000,000 

$15,345,000 

$8,661,851 

7 

so 

$1,235,000 

$50,000 

$60,000 

$5,000,000 

$9,000,000 

$15,345,000 

$7,874,411 

8 

so 

$1,235,000 

$50,000 

$60,000 

$5,000,000 

$9,000,000 

$15,345,000 

$7,158,556 

9 

so 

$1,235,000 

$50,000 

$60,000 

$5,000,000 

$9,000,000 

$15,345,000 

$6,507,778 

10 

so 

$1,235,000 

$50,000 

$60,000 

$5,000,000 

$9,000,000 

$15,345,000 

$5,916,162 

$138,000 

$12,350,000 

$1,450,000 

$1,140,000 

$45,000,000  $81,000,000  $141,078,000 

$83,041,110 
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SUMMARY  AMD  6DIDELIME8 

The  previous  section  presented  results  obtained  by 
administering  the  DEFT,  administering  the  RDIA,  and  determining 
cost  estimates  for  each  of  the  targeted  R&D  programs.  This 
section  summarizes  the  results  and  refers  back  to  the  Decision 
Rules  For  Comparing  Alternatives,  Table  1  (p.2),  to  describe  how 
the  principal  investigator  or  the  sponsor  can  use  the  obtained 
information.  Also,  guidelines  are  presented  for  choosing  the 
appropriate  training  effectiveness  and  cost  measures  for  an 
impact  analysis. 

DATA  SUMMARY 

Table  19  presents  a  summary  of  the  results  for  the  four  R&D 
programs.  In  order  to  make  decisions  using  summative  bottom 
line  data,  one  needs  to  employ  the  rationale  in  Table  1,  p.2. 
For  example,  the  Marksmanship  Training  Simulator  (MTS)  received  a 
low  DEFT  score,  a  high  RDIA  score,  and  the  life  cycle  cost  was 
estimated  at  $90M.  Both  the  DEFT  and  the  RDIA  indicated  that  the 
MTS  would  be  more  training  effective  than  the  alternative  method 
of  training  with  live  fire;  and,  the  MTS  was  estimated  as  having 
a  different  life  cycle  cost  ($90M  vs  $141M) .  Referring  to  Table 
1,  the  summative  data  put  the  MTS  in  the  third  case;  i.e., 
different  training  effectiveness  and  different  life  cycle  cost. 
The  rule  from  Table  1  states  that  in  this  circumstance  the 
alternative  with  the  higher  training  effectiveness  and  the  lower 
life  cycle  cost  should  be  selected.  Consequently,  tae  MTS  should 
be  chosen  over  the  alternative  method  of  training  with  live  fire. 
This  rationale  can  similarly  be  applied  to  the  results  of  the 
other  R&D  programs. 

Selecting  the  appropriate  methodology  for  obtaining 
estimates  of  training  effectiveness  and  costs  is  one  of  the  most 
important  initial  steps  that  a  principal  investigator  will  take 
when  performing  an  impact  analysis.  To  obtain  training 
effectiveness  estimates,  both  the  DEFT  and  the  RDIA  were  applied 
to  each  program  in  this  report.  This  was  not  necessary  since 
both  instruments  produce  measures  of  training  effectiveness. 
This  was  done  pedagogically  to  provide  examples  for  demonstrating 
how  an  impact  analysis  can  .  be  performed.  The  remainder  of  this 
section  proposes  guidelines  for  selecting  the  appropriate 
training  effectiveness  measure  and  provides  additional 
considerations  for  an  impact  analysis. 
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Table  19 


Summary  Results:  R&D  Project  vs  Alternative 
Compared  on  Effectiveness  and  Cost 


R&D  PROGRAM 

_ ALTERNATIVE _ 

Training 

Effectiveness 

Life  Cycle 

Cost  S 

Decision 

Embedded  Training 

<  Embedded 
Training 

Fleet  Exercise, 

Fuel,  A/C  Flight 
Training  (Assume 
>  Embedded  Training) 

Subjective  - 
Select  if 
funding  is 
available 

Passive  Acoustic 
Analysis  Trainer 

<  PAAT 

>  PAAT 

Select 

PAAT 

Marksmanship 
Training  Simulator 

<  MTS 

>  MTS 

Select 

MTS 

Trainer  for  Radar 
Intercept  Operator 

<  TRIO 

>  TRIO 

Select 

TRIO 
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As  stated  previously,  R&D  programs  differ  on  many  factors 
such  as  purpose,  scope,  cost,  guantif iability  of  results,  etc. 
Some  R&D  programs  are  intended  for  demonstrating  a  training 
phenomenon,  while  others  actually  train;  or,  some  programs 
involve  an  Improved  feature  for  an  existing  trainer;  or,  other 
progreuBS  are  concerned  with  refresher  training.  Moreover,  in 
some  programs  the  training  requirement  may  be  better  defined  than 
in  others.  Therefore,  prior  to  selecting  an  instrument  which 
measures  training  effectiveness,  the  principal  investigator 
should  first  determine  which  stage  of  development  the  program  is 
in  and  also  the  kinds  of  data  which  are  available. 

There  are  a  number  of  instruments,  either  in  the  literature 
or  developed  for  a  specific  application,  for  measuring  training 
effectiveness  which  are  similar  to  the  DEFT  and  the  RDIA.  These 
two  were  chosen  for  their  versatility,  flexibility,  and 
applicability.  Instruments  such  as  the  DEFT  should  be  chosen  if 
the  R&D  program  actually  trains  or  if  the  R&D  program's  training 
requirement  is  well  defined.  Instruments  like  the  RDIA  are 
recommended  if  the  R&D  program  involves  an  improvement  to  an 
existing  training  system,  if  the  proposed  training  research  is 
intended  only  as  a  demonstration,  or  if  the  proposed  training 
system  is  intended  to  be  used  for  refresher  training. 

Once  a  particular  approach/methodology  has  been  determined, 
questions  may  arise  concerning  who  identifies  program  attributes 
to  be  measured,  who  determines  the  importance  of  these 
attributes,  and  who  rates/ranks  the  alternatives  on  the 
identified  attributes.  Although  much  depends  on  the  progress  and 
status  of  the  research  program,  the  following  are  recommended: 

1.  The  principal  investigator  identifies  the  program 
attributes  which  should  be  measured. 

2 .  The  sponsor  and/or  ultimate  users  determine  the 
importance  of  the  identified  attributes;  i.e.,  weigh  the 
importance  of  the  attributes. 

3.  The  ultimate  user  rates/ranks  the  alternatives  based  on 
the  attributes;  however,  the  principal  investigator  may  be  the 
best  or  only  judge  available. 

4.  Choose  from  among  cost  categories  or  variables  those 
which  capture  as  much  as  possible  the  cost  of  operating  the 
status  quo  alternative  as  well  as  the  cost  of  developing  and 
implementing  the  R&D  product. 
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In  closing,  the  principles  set  forth  in  this  report  are  offered 
only  as  guidelines  which  a  princip-*!  investigator  can  use  to 
provide  estimates  of  cost  and  effectiveness  to  a  sponsor  or 
potential  sponsor  which  would  give  them  reliable  information  for 
determining  the  outcome  of  a  program/project.  Even  though  R&D 
programs  may  have  similar  characteristics,  each  program  should  be 
examined  closely  and  treated  uniquely  within  the  recommended 
methodologies  proposed  here.  Important  funding  decisions  have  to 
be  made.  Therefore,  the  onus  is  on  the  principal  investigator  to 
provide  the  best  information  he  or  she  can  obtain  to  support 
rational  decisions. 
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DEVICE  EFFECTIVENESS  FORECASTING  TECHNIQUE  (DEFT) 


1.  Considering  what  you  know  about  the  following  trainee's 
background,  work  experience,  and  prior  training,  what 
proportion  of  skills  and  knowledges  required  to  meet  the 
training  objective(s)  will  the  trainee  still  have  to  learn  to 
reach  criterion  proficiency  for  each  dimension  shown  below. 

OsNone;  the  trainee  can  already  use  this  dimension  of 
the  device 


A-school  Pre-AIC  Post-AIC 

Grad  Grad  Grad 

Dimension  Ratine  Rating  Rating 

Visual  Display  _  _  _ 

Manual  Control  _  _  _ 

Communication 


100=A11;  the  trainee  has  to  learn  to  use  this  dimension 
of  the  device 


2.  Consider  each  task  that  a  trainee  won't  be  able  to 
perform  initially  using  Embedded  Training.  Rate  the 
difficulty  the  trainee  will  have  in  using  each  dimension  of 
Embedded  Training. 

l=Very  easy  to  learn;  it  will  take  practically  no 
training  or  practice  to  reach  criterion  proficiency 


on  this  dimension 

A-school 

Pre-AIC 

Post-AIC 

Grad 

Grad 

Grad 

Dimension 

Rating 

Rating 

Rating 

Visual  Display 
Manual  Control 
Communication 

100=Very  difficult  to  learn;  it  will 

take  a 

lot  of 

training  or  practice  to  reach  criterion  proficiency 
on  this  dimension 
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3.  Consider  the  performance  deficits  you  have  identified 
(see  item  1).  Rate  how  well  Embedded  Training  will  overcome 
these  deficits. 


l«Poor  training  of  air  intercept  tasks 


Dimension 


A-school 

Grad 

Rating 


Pre-AIC 

Grad 

Rating 


Post-AIC 

Grad 

Rating 


Visual  Display 
Manual  Control 
Communication 


lOOsGood  training  of  air  intercept  tasks 


4.  How  well  does  each  dimension  of  Embedded  Training  provide 
practice? 


l=Little  practice  is  provided 


Dimension 


A-school 

Grad 

Rating 


Pre-AIC 

Grad 

Rating 


Post-AIC 

Grad 

Rating 


Visual  Display 
Manual  Control 
Communication 


100=Extensive  practice  is  provided 


5.  For  each  dimension,  does  Embedded  Training  provide 
qualitative  feedback  to  the  trainee? 


0*Feedback  is  not  provided 

A-school  Pre-AIC 
Grad  Grad 

Dimension  Rating  Rating 


Post-AIC 

Grad 

Rating 


Visual  Display 
Manual  Control 
Communication 


lOOaLots  of  feedback  is  provided 
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6.  Does  Embedded  Training  provide  a  record  of  trainee 
performance  when  using  the  dimensions  shown  below? 

0»No  record  of  trainee  performance  is  provided 


A-school 

Pre-AIC 

Post-AIC 

Grad 

Grad 

Grad 

Dimension 

Rating 

Rating 

Rating 

Visual  Display 
Manual  Control 
Communication 


100=Complete  record  of  trainee  performance  is  provided 

7.  Consider  each  dimension  of  Embedded  Training  you  feel  the 
trainee  won't  be  able  to  use  initially  on  the  AN/SPA-25G. 

Rate  the  difficulty  the  trainee  will  have  in  learning  to  use 
each  dimension. 

l=Very  easy  to  learn;  it  will  take  practically  no 
training  or  practice  on  the  AN/SPA-25G  to  reach 
criterion  proficiency  on  this  dimension 


Dimension 


A-school 

Grad 

Rating 


Pre-AlC 

Grad 

Rating 


Post-AIC 

Grad 

Rating 


Visual  Display 
Manual  Control 
Communication 


lOOsVery  difficult  to  learn;  it  will  take  a  lot  of 
training  or  practice  on  the  AN/SPA-25G  to  reach 
criterion  proficiency  on  this  dimension 
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8.  Physical  similarity  Is  based  on  the  similarity  between 
physical  characteristics  of  training  and  those  of  the 
operational  situation.  Rate  the  physical  similarity  between 
the  AN/SPA-25G  and  its  counterpart  in  Embedded  Training. 


OaNo  similarity;  although  the  dimension  is  represented 
in  Embedded  Training,  there  would  be  a  large  noticeable 
difference  quite  apparent  to  the  trainee  at  transfer 
and  a  large  performance  decrement,  given  that  the 
trainee  could  use  the  dimension  at  all;  specific 
instruction  and  practice  would  be  required  for  this 
dimension  on  the  AN/SPA-25G  after  transfer  to 
overcome  the  deficit. 

A-school  Pre-AIC  Post-AIC 
Grad  Grad  Grad 

Dimension  Rating  Rating  Rating 

Visual  Display  _  _  _ 

Manual  Control  _  _  _ 

Communication  _ 


AAWW  WW  Vt«WWA«  wm  M  w  V*  S*  W  W  AA  i*A*S.* 


C*  ^  ^  W' 


9.  Functional  similarity  is  based  on  the  operator’s  behavior 
in  terms  of  the  information  flov;  from  auditory  and/or  visual 
displays  to  the  operator  and  from  the  operator  to  each 
control.  Rate  the  functional  similarity  for  each  dimension 
on  the  AN/SPA-25G  and  its  counterpart  (if  any)  in  Embedded 
Training. 


0=No  similarity;  for  this  dimension  the  trainee  acts  on 
completely  different  amounts  and  types  of  information 
in  the  Embedded  Training  situation  and  the  AN/SPA-25G 


Dimension 

A-school 

Grad 

Rating 

Pre-AIC 

Grad 

Rating 

Post-AIC 

Grad 

Rating 

Visual  Display 

Manual  Control 

Communication 
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lOOsIdentical :  for  this  dimension  the  trainee  acts  on 
the  same  types  and  amounts  of  information  in 
Embedded  Training  and  the  AN/SPA-25G 


10.  Consider  the  statement  of  the  operational  performance 
objective(s) ,  the  training  ob jective(s) ,  and  descriptions  of 
the  AN/SPA-25G  and  Embedded  Training.  Also,  consider  the 
instructional  features  and  training  principles  that  are 
included  in  Embedded  Training  to  increase  the  probability 
that  the  skills  and  knowledges  acquired  will  be  used 
effectively  on  the  AN/SPA-25G.  For  each  dimension,  rate  how 
well  Embedded  Training  will  promote  transfer  to  the 
AN/SPA-25G. 

0=Dimension  is  not  realistic,  relevant  or  similar  to 
the  AN/SPA-25G 


Dimension 

Visual  Display 
Manual  Control 
Communication 


A-school  Pre-AIC  Post-AIC 


Grad 


Grad 


Rating  Rating 


Grad 

Rating 


100=Dimension  is  relevant,  realistic,  or  similar  to  the 
AN /SPA-2 5G 


11.  For  the  dimensions  that  must  be  used  in  Embedded 
Training,  are  the  conditions  of  practice  occurring  late  in 

M  ^  A  ^  V  A  ^  V*  , 

UaO  .  . . .  ”  *  * 


. 


AliT/CDA  'yc  n  o 
1  -  ••  rvif/  • 


v**yXAu€ii0^vii  Hvw  oy  w/ AXiuo  wc  i\i%/ 


A-school 
Cx*  sd 
Rs wxng 


Pre-AIC 
Gr  ad 


X>r\  *5^  _  A  m 
irOo»»— 

Gr  sd 
Rating 


Visual  Display 
Manual  Control 
Communication 


lOOsDimension  approximates  the  AN/SPA-25G  very  well 
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12.  For  each  dimension  that  must  be  used  in  the  AN/SPA-25G, 
is  an  extensive  amount  of  practice  given  in  Embedded 
Training? 

0»Not  practiced  in  Embedded  Training 


A-school  Pre-AIC  Post-AIC 

Grad  Grad  Grad 

Dimension  Rating  Rating  Rating 


Visual  Display 
Manual  Control 
Communication 


lOOsPracticed  extensively  in  Embedded  Training 
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DEVICE  EFFECTIVENESS  FORECASTING  TECHNIQUE  (DEFT) 


1.  Considering  what  you  know  about  the  typical  Naval  Flight 
Officer’s  background,  work  experience,  and  prior  training, 
what  proportion  of  skills  and  knowledges  required  to  meet  the 
training  objective(s)  will  the  trainee  still  have  to  learn  to 
reach  proficiency  on  the  trainer  for  radar  intercept  officers 
(TRIO)  for  each  of  the  dimensions  shown  below. 

OsNone;  the  trainee  can  already  use  this  dimension  of 
the  device 


Dimension  Rating 

Visual  display  _ 

Audio  present  . 

Oral  communication _ 

100=A11;  the  trainee  has  to  learn  to  use  this 
dimension  of  the  device 


2.  Consider  each  air  intercept  task  that  a  trainee  won’t  be 
able  to  perform  initially  on  TRIO.  Rate  the  difficulty  the 
typical  Naval  Plight  Officer  will  have  in  using  each 
dimension  of  TRIO. 

l=Very  easy  to  learn;  it  will  take  practically  no 
training  or  practice  to  reach  criterion  proficiency 
on  this  dimension 


Dimension  Rating 

Visual  display  _ 

Audio  present  _ 

Oral  communication _ 

100=Very  difficult  to  learn;  it  will  take  a  lot  of 
training  or  practice  to  reach  criterion 
proficiency  on  this  dimension 
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3.  Consider  the  performance  deficits  you  identified  (see 
item  1).  Rate  how  well  using  TRIO  will  overcome  these 
deficits? 

l*Poor  training  of  radar  intercept  officer  tasks 

Dimension  Rating 


Visual  display 
Audio  present 
Oral  communication 


lOOsGood  training  of  radar  intercept  officer  tasks 


How  well  does  each  dimension  of  TRIO  provide  practice 
on  air  intercept  tasks? 

l=Little  practice  is  provided 

Dimension  Rating 


Visual  display 
Audio  present 
Oral  communication 

lOOxExtensive  practice  is  provided 


5.  For  each  dimension  that  is  used,  does  TRIO  provide 
qualitative  feedback  to  the  trainees  about  their  performance? 

OxFeedback  not  provided 


Dimension 


Rating 


Visual  display 
Audio  present 
Oral  communication 

100=Lots  of  feedback  provided 
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6,  Does  TRIO  provide  a  record  of  trainee  performance  when 
using  the  dimensions  shown  below? 

0*No  record  of  trainee  performance  is  provided 

Dimension  Rating 

Visual  display  _ 

Audio  present  _ 

Oral  communication _ 

lOOsCompleted  record  of  trainee  performance  is 
provided 


7.  Consider  each  dimension  of  TRIO  you  feel  the  typical 
trainee  won't  be  able  to  use  initially  on  the  F-14.  Rate  the 
difficulty  the  typical  student  Naval  Flight  Officer  will  have 
in  learning  to  use  each  dimension. 

IxVery  easy  to  learn;  it  will  take  practically  no 
training  or  practice  on  the  F-14  to  reach  criterion 
proficiency  on  this  dimension 

Dimension  of 

actual  equip  Rating 

Visual  display  _ 

Audio  present  _ 

Oral  communication  _ 


100=Very  difficult  to  learn;  it  will  take  a  lot  of 
training  or  practice  on  the  F-lU  to  reach 
criterion  proficiency  on  this  dimension 
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8.  Physical  similarity  is  based  on  the  similarity  between 
physical  characteristics  of  the  training  and  those  of  the 
F-14.  Rate  the  physical  similarity  between  the  F-1^  and  its 
counterpart  on  the  TRIO. 

OsNo  similarity:  although  the  dimension  is 

represented  in  the  TRIO,  there  would  be  a  large 
noticeable  difference  quite  apparent  to  the  trainee 
at  transfer  and  a  large  performance  decrement, 
given  that  the  trainee  could  use  the  dimension  at 
all;  specific  instruction  and  practice  would  be 
required  for  this  dimension  on  the  F-14  after 
transfer  to  overcome  the  deficit 

Dimension  Rating 


Visual  display  _ 

Audio  present  _ 

Oral  communication _ 

lOOsIdentical ;  the  trainee  would  not  notice  a 

difference  between  the  TRIO  and  the  F-14  for  this 
dimension  at  the  time  of  transfer 


9.  Functional  similarity  is  based  on  the  operator's  behavior 
in  terms  of  the  information  flow  from  auditory  and  visual 
displays  to  the  operator  and  from  the  operator  to  each 
control.  Rate  the  functional  similarity  for  each  dimension 
on  the  F-14  and  its  counterpart  (if  any)  on  the  TRIO. 

0=No  similarity;  for  this  dimension  the  trainee  acts 
on  completely  different  amounts  and  types  of 
information  on  the  TRIO  and  the  F-l4 

Dimension  Rating 


Visual  display  _ 

Audio  present  _ 

Oral  communication _ 

100*Identlcal ;  for  this  dimension  the  trainee  acts  on 
the  same  types  and  amounts  of  information  on  the 
TRIO  and  the  F-14 


A-11 
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10.  Consider  the  operational  performance  oh jective{s) ,  the 
training  objective(s) ,  and  descriptions  of  the  F-14  and  the 
TRIO.  Also,  consider  the  instructional  features  and  training 
principles  that  are  included  in  the  TRIO  to  increase  the 
probability  that  the  skills  and  knowledges  acquired  will  be 
used  effectively  on  the  F-1^.  For  each  dimension,  rate  how 
well  TRIO  will  promote  transfer  of  training  to  the  F-l^J. 

OsDimension  is  not  realistic,  relevant  or  similar  to 
the  F-14 


Dimension  Rating 


Visual  display  _ 

Audio  present  _ 

Oral  communication _ 

100=Dimension  is  relevant,  realistic,  or  similar  to 
the  F-14 


11.  For  dimensions  that  must  be  used  in  TRIO,  will  the 
conditions  of  practice  occurring  late  in  training  be  made  to 
approximate  those  in  the  F-14? 

OsDimension  does  not  approximate  the  F-14 

Dimension  Rating 


Visual  display 
Audio  present 
Oral  communication 

lOOsDimension  approximates  the  F-14  very  well 
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12.  For  the  dimensions  used  in  the  F-14,  is  an  extensive 
amount  of  practice  given  in  the  TRIO? 

0=Not  practiced  in  the  TRIO 


Dimension  Rating 


Visual  display 
Audio  present 
Oral  communication 


100=Practiced  extensively 


WiiC 
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DEVICE  EFFECTIVENESS  FORECASTING  TECHNIQUE  (DEFT) 


1.  Considering  what  you  know  about  the  typical  trainee's 
background,  work  experience,  and  prior  training,  what 
proportion  of  skills  and  knowledges  required  to  meet  the 
training  objective(s)  will  the  trainee  still  have  to  learn  to 
reach  criterion  proficiency  on  the  Passive  Acoustic  Analysis 
Trainer  (PAA)  for  each  dimension  shown  below. 

0=None;  the  trainee  can  already  use  this 
the  device 

Dimension 


Visual  Display 
Audio  Present 
Manual  Control 
Communication 


lOOsAll;  the  trainee  has  to  learn  to  use 
dimension  of  the  device 


2.  Consider  each  gram  analysis  that  a  trainee  won't  be  able 
to  perform  initially  on  the  PAA  trainer.  Rate  the  difficulty 
the  typical  trainee  will  have  in  using  each  dimension  of  the 
device . 


dimension  of 

Rating 


this 


l=Very  easy  to  learn;  it  will  take  practically  no 
training  or  practice  to  reach  criterion  proficiency 
on  this  dimension 


Dimension  Rating 

Visual  Display  _ 

Audio  Present  _ 

Manual  Control  _ 

Communication 


100=Very  difficult  to  learn;  it  will  take  a  lot  of 
training  or  practice  to  reach  criterion 
proficiency  on  this  dimension 
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3.  Consider  the  performance  deficits  (see  item  1)  you  have 
identified.  For  each  dimension,  rate  how  well  using  the  PAA 
trainer  will  overcome  these  deficits. 

l*Poor  training  of  gram  analysis 

Dimension  Rating 

Visual  Display  _ 

Audio  Present  _ 

Manual  Control  _ 

Communication 


100=Gooq  training  of  gram  analysis 


4.  How  well  does  each  dimension  of  the  PAA  trainer  provide 
practice? 

0=Little  practice  is  provided 

Dimension  Rating 

Visual  Display  _ 

Audio  Present  _ 

Manual  Control  _ 

Communication 


100=Extensive  practice  is  provided 


5.  For  each  dimension  that  is  used,  does  the  PAA  trainer 
provide  qualitative  feedback  to  the  trainee? 


OsFeedback  not  provided 

Dimension  Rating 

Visual  Display  _ 

Audio  Present  _ 

Manual  Control  _ 

Communication  _ 


100=Lots  of  feedback  provided 
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6.  Does  the  PAA  trainer  provide  a  record  of  trainee 
performance  when  using  the  dimensions  shown  below? 

0«No  record  of  trainee  performance  is  provided 

Dimension  Rating 

Visual  Display  _ 

Audio  Present  _ 

Manual  Control  _ 

Communication  _ 


lOOaComplete  record  of  trainee  performance  is 
provided 


7.  Consider  each  dimension  of  the  PAA  trainer  you  feel  the 
typical  traihee  won't  be  able  to  use  initially  on  the  BQQ-3. 
Rate  the  difficulty  the  typical  trainee  will  have  in  learning 
to  use  each  dimension. 

l=Very  easy  to  learn;  it  will  take  practically  no 
training  or  practice  on  the  BQQ-3  to  reach 
criterion  proficiency  on  this  dimension 

Dimension  Rating 

Visual  Display  _ 

Audio  Present  _ 

Manual  Control  _ 

Communication  _ 


lOOsVery  difficult  to  learn;  it  will  take  a  lot  of 
training  or  practice  on  the  BQQ-3  to  reach 
criterion  proficiency  on  this  dimension 
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8.  Physical  similarity  is  based  on  the  similarity  between 
physical  characteristics  of  the  training  device  and  those  of 
the  BQQ-3.  Rate  the  physical  similarity  between  the  BQQ-3 
and  its  counterpart  on  the  PAA  trainer. 

OsNo  similarity:  although  the  dimension  is 

represented  in  the  PAA  trainer,  there  would  be  a 
large  noticeable  difference  quite  apparent  to  the 
trainee  at  transfer  and  a  large  performance 
decrement,  given  that  the  trainee  could  use  the 
dimension  at  all;  specific  instruction  and  practice 
would  be  required  for  this  dimension  on  the  BQQ-3 
after  transfer  to  overcome  the  deficit 

Dimension  Rating 

Visual  Display  _ 

Audio  Present  _ 

Manual  Control  _ 

Communication 


100=Identical ;  the  trainee  would  not  notice  a 
difference  between  the  PAA  trainer  and  the 
BQQ-3  for  this  dimension  at  the  time  of  transfer 


9.  Functional  similarity  is  based  on  the  operator's  behavior 
in  terms  of  the  information  flow  from  auditory  and  visual 
displays  to  the  operator  and  from  the  operator  to  each 
control.  Rate  the  functional  similarity  for  each  dimension 
on  the  BQQ-3  ano  its  counterpart  Cif  any)  on  the  PAA  trainer. 

0=No  similarity;  for  this  dimension  the  trainee  acts 
on  completely  different  amounts  and  types  of 
information  in  the  PAA  trainer  and  BQQ-3 

Dimension  Rating 

Visual  Display  _ 

Audio  Present  _ 

Manual  Control  _ 

Communication  _ 


lOOsIdentical ;  for  this  dimension  the  trainee  acts  on 
the  same  types  and  amounts  of  information  in  the 
PAA  trainer  and  the  BQQ-3 
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10.  Consider  the  statement  of  the  operational  ps 
oh jective(s) ,  the  training  oh jeative(s) ,  and  descriptions  of 
the  BQQ-3  and  the  PAA  trainer.  Also,  consider  the 
instructional  features  and  training  principles  that  are 
included  in  the  PAA  trainer  to  increase  the  probability  that 
the  skills  and  knowledges  acquired  will  be  used  effectively 
in  the  BQQ-3.  For  each  dimension,  rate  how  well  the  PAA 
trainer  will  promote  transfer  to  the  BQQ-3. 


OsDimension  is  not  realistic,  relevant,  or  similar  to 
the  BQQ-3 


Dimension  Rating 

Visual  Display  _ 

Audio  Present  _ 

Manual  Control  _ 

Communication  _ 


100=Dimension  is  relevant,  realistic,  or  similar  to 
the  BQQ-3 


11.  For  dimensions  that  must  be  used  in  the  PAA  trainer,  are 
the  conditions  of  practice  occurring  late  in  training  made  to 
approximate  those  in  the  BQQ-3? 

0=Dimension  does  not  approximate  the  BQQ-3 

Dimension  Rating 


Visual  Display 
Audio  Present 
Manual  Control 
Communication 


lOOsDimension  approximates  the  BQQ-3  very  well 
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12.  For  dimensions  that  must  be  used  in  the  BQQ-3.  is  an 
extensive  amount  of  practice  given  in  the  PAA  trainer? 

0*Not  practiced  in  the  PAA  trainer 

Dimension  Rating 


Visual  Display 
Audio  Present 
Manual  Control 
Communication 


lOOaPracticed  extensively  in  the  PAA  trainer 
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DEVICE  EFFECTIVENESS  FORECASTING  TECHNIQUE  (DEFT) 


1.  Considering  what  you  know  about  the  typical  recruit's 
background,  work  experience,  and  prior  training,  what 
proportion  of  skills  and  knowledges  required  to  meet  the 
training  objective(s)  will  the  recruit  still  have  to  learn  to 
reach  criterion  proficiency  for  each  dimension  shown  below. 

0«None;  the  recruit  can  already  use  this  dimension  of 
the  device 


Dimension  Rating 

Visual  display  _ 

Audio  present  _ 

Manual  control 


lOOsAll;  the  recruit  has  to  learn  to  use  this 
dimension  of  the  device 


2.  Consider  each  marksmanship  task  that  a  recruit  won't  be 
able  to  perform  initially  on  the  marksmanship  trainer  (MET). 
Rate  the  difficulty  the  typical  recruit  will  have  in  using 
each  dimension  of  the  trainer. 

l=Very  easy  to  learn;  it  will  take  practically  no 
training  or  practice  to  reach  criterion  proficiency 
on  this  dimension 

Dimer.sion  Rating 


Visual  display 
Audio  present 
Manual  control 


lOOsVery  difficult  to  learn;  it  will  take  a  lot  of 
training  or  practice  to  reach  criterion 
proficiency  on  this  dimension 
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3.  Consider  the  performance  deficits  you  have  identified 
(see  item  1).  Rate  how  well  using  the  MET  will  overcome 
these  deficits. 

IsPoor  training  of  marksmanship 

Dimension  Rating 

Visual  display  _ 

Audio  Present  _ 

Manual  control 


100=Good  training  of  marksmanship 


How  well  does  each  dimension  of  the  MET  provide 
practice? 

l=Little  practice  is  provided 

Dimension  Rating 


Visual  display 
Audio  Present 
Manual  control 


100=Extensive  practice  is  provided 


5.  For  each  dimension  that  is  used,  does  the  MET  provide 
qualitative  feedback  to  the  trainees? 

0=Feedback  not  provided 


Dimension 


Rating 


Visual  display 
Audio  present 
Manual  control 


lOOsLots  of  feedback  provided 
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6.  Does  the  MET  provide  a  record  of  trainee  performance  when 
using  the  dimensions  shown  below? 

OaNo  record  of  trainee  performance  is  provided 

Dimension  Rating 

Visual  display  _ 

Audio  present  _ 

Manual  control  _ 

100=Complete  record  of  trainee  performance  is 
provided 


7.  Consider  each  dimension  of  the  MET  you  feel  the  typical 
recruit  won't  be  able  to  use  initially  using  "live  fire." 

Rate  the  difficulty  the  typical  recruit  will  have  in  learning 
to  use  each  dimension. 

l=Very  easy  to  learn;  it  will  take  practically  no 
training  or  practice  when  using  "live  fire"  to 
reach  criterion  proficiency  on  this  dimension 


Dimension  of 
actual  equip 


Visual  display 
Audio  present 
Manual  control 


100*Very  difficult  to  learn;  it  will  take  a  lot  of 
training  or  practice  when  using  "live  fire"  to 
reach  criterion  proficiency  on  this  dimension 
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8.  Physical  similarity  is  based  on  the  similarity  between 
physical  characteristics  of  the  training  device  and  those  of 
the  actual  situation.  Rate  the  physical  similarity  between 
using  "live  fire"  and  its  counterpart  on  the  MET. 

0=Mo  similarity;  although  the  dimension  is 

represented  in  the  MET,  there  would  be  a  large 
noticeable  difference  quite  apparent  to  the  recruit 
at  transfer  and  a  large  performance  decrement, 
given  that  the  recruit  could  use  the  dimension  at 
all;  specific  instruction  and  practice  would  be 
required  for  this  dimension  when  using  "live  fire" 
after  transfer  to  overcome  the  deficit 

Dimension  Rating 

Visual  display  _ 

Audio  present  _ 

Manual  control 


100=Identical ;  the  recruit  would  not  notice  a 
difference  between  the  MET  and  using  "live 
fire"  for  this  dimension  at  the  time  of  transfer 


9.  Functional  similarity  is  based  on  the  operator's  behavior 
in  terms  of  the  information  flow  from  auditory  and  visual 
displays  to  the  operator  and  from  the  operator  to  each 
control.  Rate  the  functional  similarity  for  each  dimension 
using  "live  fire"  and  its  counterpart  (if  any)  on  the  MET. 

OaNo  similarity;  for  this  dimension  the  recruit  acts 
on  completely  different  amounts  and  types  of 
information  in  the  MET  and  using  "live  fire" 

Dimension  Rating 

Visual  display  _ 

Audio  present  _ 

Manual  control 


lOOaldentical ;  for  this  dimension  the  recruit  acts  on 
the  same  types  and  amounts  of  information  in  the 
MET  and  using  "live  fire" 
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10.  Consider  the  statement  of  the  operational  performance 
objective(s) ,  the  training  objective(s)  and  descriptions  of 
the  NET.  Also,  consider  the  instructional  features  and 
training  principles  that  are  included  in  the  MET  to  Increase 
the  probability  that  the  skills  and  knowledges  acquired  will 
be  used  effectively  when  using  "live  fire."  For  each 
dimension,  rate  how  well  the  MET  will  promote  transfer  to  the 
"live  fire"  situation. 

0*Dimension  is  not  realistic,  relevant  or  similar  to 
the  "live  fire"  situation 


Dimension  Rating 


Visual  display 
Audio  present 
Manual  control 


100=Dimension  is  relevant,  realistic,  or  similar  to 
the  "live  fire"  situation 


11.  For  dimensions  that  must  be  used  in  the  MET,  are  the 
conditions  of  practice  occurring  late  in  training  made  to 
approximate  those  in  the  "live  fire"  situation? 

OsDimension  does  not  approximate  the  "live  fire" 
situation 


Dimension 


Rating 


Visual  display 
Audio  present 
Manual  control 

lOOsDimension  approximates  the  "live  fire" 
situation  very  well 
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12.  For  dimensions  that  must  be  used  in  the  "live  fire” 
situation,  is  an  extensive  amount  of  practice  given  in  the 
MET? 

0«Not  practiced  in  the  MET 


Dimension  Rating 


Visual  display 
Audio  present 
Manual  control 

lOOsPracticed  extensively  in  the  MET 
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APPENDIX  B 
RDIA  QUESTIONNAIRES 


B-1 


Technical  Report  88-017 


EVALUATION  OP  EMBEDDED  TRAINING 


When  answering  the  following  items,  compare  the  current 
training  system/procedures  with  the  addition  of  the  proposed 
RDT&E  training  system/procedures  (i.e..  Embedded  Training). 
Indicate  what  changes  in  the  following  factors  will  occur 
with  the  implementation  of  the  RDT&E  system/procedures 
(Please  circle  one  number). 


Change  in; 


Training  time 


1 

2 

3 

5 

Decrease 

No 

change 

Increase 

Required  instructor 

time 

1 

2 

3 

k 

5 

Decrease 

No 

change 

Increase 

Training  content 

1 

2 

3 

5 

Decrease 

No 

Change 

Increase 

User  acceptance 

1 

2 

3 

4 

5 

Decrease 

No 

change 

Increase 

Instructor 

acceptance 

1 

2 

3 

4 

5 

Decrease 

No 

change 

Increase 

Availability  to 

practice 

1 

2 

3 

4 

5 

Decrease 

No 

change 

Increase 

Feedback  to 

trainee 

1 

2 

3 

4 

5 

Decrease 

No 

change 

Increase 
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Student  performance: 

speed  1  2 

Decrease 


3 

No  change 


Student  performance: 
accuracy 


M  5 

Increase 

4 


Additional  comments 


1 

Decrease 


3 

No  change 


5 

Increase 
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EVALUATION  OF  TRAINER  FOR  RADAR  INTERCEPT  OFFICERS 


When  answering  the  following  items,  compare  the  current 
training  system/procedures  with  the  addition  of  the  proposed 
RDT&E  training  system/procedures  (i.e.,  TRIO).  Indicate  what 
changes  in  the  following  factors  will  occur  with  the 
implementation  of  the  RDT&E  system/procedures  (Please  circle 
one  number). 


Change  In; 


Training  tim^ 


Required  instructor 
time 


Trainin'’  content 


User  acceptance 


Instr'_ct  or 


Availability  to 
practice 


Feedback  to 
trainee 


1  2 
Decrease 


1  2 

Decrease 

1  2 

Decrease 

1  2 

Decrea  se 

1  2 

Decrease 


1  2 
Decrease 


1  2 
Decrease 


No  chan^’e  Increase 


3  4  5 

No  change  Increase 


No  chan I’e 


No  change 


a  c; 

T nr'T’oa  ap 


4  5 

T nnr'pa  «io 


^  a  ^ 

No  change  Increase 


No  change 


No  change 


4  5 

Increase 


4  5 

Increase 
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Student 

speed 

performance : 

1  2 
Decrease 

No 

3 

change 

Student  performance: 
accuracy 

1  2 
Decrease 

No 

3 

change 

4 


5 

Increase 

5 


Increase 
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EVALUATION  OF  PASSIVE  ACOUSTIC  ANALYSIS  TRAINER 


When  answering  the  following  items,  compare  the  current 
training  system/procedures  (i.e.,  BQQ-3)  with  the  proposed 
RDT&E  training  system/procedures  (i.e.,  PAA  trainer). 
Indicate  what  changes  in  the  following  factors  will  occur 
with  the  implementation  of  the  RDT&E  system/procedures 
(Please  circle  one  number). 


Change  in: 


Training  time 


Decrease 


3^5 
No  change  Increase 


Required  instructor 
time 


Decrease 


No  change 


^  5 

Increase 


Training  content 


Decrease 


No  change 


^  5 

Increase 


User  acceptance 


Decrease 


No  change 


^  5 

Increase 


Instructor 

acceptance 


Decrease 


No  change 


5 

Increase 


Availability  to 
practice 


Decrease 


No  change 


^  5 

Increase 


Feedback  to 
trainee 


Decrease 


No  change 


^  5 

Increase 
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• 

Student  performance: 

speed 

1  2 

3 

4 

5 

Decrease 

No 

change 

Increase 

Student  performance: 

accuracy 

1  2 

3 

4 

5 

Decrease 

No 

change 

Increase 

Additional  comments  _ 
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EVALUATION  OP  MARKSMANSHIP  EXPERT  TRAINER 


When  answering  the  following  items,  compare  the  current 
training  system/procedures  (i.e.,  live  fire)  with  the 
proposed  RDT&E  training  system/procedures  (i.e.,  MET). 
Indicate  what  changes  in  the  following  factors  will  occur 
with  the  implementation  of  the  RDT&E  system/procedures 
(Please  circle  one  number). 


Change  in; 


Training  time 


1 

2 

3 

5 

Decrease 

No 

change 

Increase 

Required  instructor 

time 

1 

2 

3 

4 

5 

Decrease 

No 

change 

Increase 

Training  content 

1 

2 

3 

4 

5 

Decrease 

No 

change 

Increase 

User  acceptance 

1 

2 

3 

4 

5 

Decrease 

No 

change 

Increase 

Instructor 

acceptance 

1 

2 

3 

4 

5 

Decrease 

No 

change 

Increase 

Availability  to 

practice 

1 

2 

3 

4 

5 

Decrease 

No 

change 

Increase 

Feedback  to 

trainee 

1 

2 

3 

4 

5 

Decrease 

No 

change 

Increase 
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Student  performance: 

speed  12  3  ^  5 

Decrease  No  change  Increase 

Student  perf oi*mance : 

accuracy  12  3  ^  5 

Decrease  No  change  Increase 


Additional  comments 
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